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Abstract 

In this paper we extend support vector machines from reproducing kernel Hilbert 
spaces into reproducing kernel Banach spaces whose reproducing kernels can be defined 
on nonsymmetric domains. Using the orthogonality of semi-inner products, we can obtain 
the empirical representations of support vector machine solutions. In addition, we can set 
up the reproduction property onto a generalized native space by Fourier transform tech- 
niques in order that it becomes a reproducing kernel Banach space and its reproducing 
kernel is given by the related positive definite function. The reproducing kernel Banach 
spaces of some reproducing kernels can be even imbedded into Sobolev spaces. We show 
some special examples of reproducing kernel Banach spaces induced by Matern functions 
(Sobolev splines) such that their support vector machine solutions are well computable 
same as the classical cases but their explicit formulas are totally different from the solu- 
tions on reproducing kernel Hilbert spaces. It is possible to produce a new numerical tool 
for support vector machines. 

Keywords: support vector machine, reproducing kernel Banach space, reproducing kernel, posi- 
tive definite function, Fourier transform, Sobolev space, Matern function, Sobolev spline. 

1 Introduction 

The theory and practice of kernel -based methods is a fast growing research area. It has been used for 
both scattered data approximation and machine learning. Their applications come from such different 
fields as physics, biology, geology, meteorology and finance. The books Pfl 171 1211 show how to use (con- 
ditionally) positive definite kernels to construct interpolants for given data sites sampled from some un- 
known functions in the native spaces corresponded to the kernel functions. In the books l2l[3l [T9ll20l . we 
can choose the optimal support vector machine solution in a reproducing kernel Hilbert space (RKHS) 
of a given reproducing kernel and this solution is introduced by the related reproducing kernel and the 
practical data values. Actually, the native spaces and the RKHSs are in the same senses and the nu- 
merical people and the statical learning people just use different languages and techniques to introduce 
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the spaces. Moreover, the recent papers J9] \TU[ |22] even develop a clear and detailed framework of 
generalized Sobolev spaces and RKHSs to make a connection with their associated Green functions and 
reproducing kernels. 

In the current research work, the papers (5] |6) extend classical native spaces to generalized native 
spaces and the papers IfTTl [THl [231 generalize RKHSs to Banach spaces. However, the reproductions of 
generalized native spaces are not discussed in 15] and IfTTl [TSl l23l do not mention how to use repro- 
ducing kernels to set up the explicit forms of the related reproducing kernel Banach spaces (RKBSs) 
defined on continuous domains, whose structure is difficult to apply into computation. (22,, Section 6] 
tries to combine these both ideas together to use the techniques of generalized native spaces to construct 
RKBSs. 

In this paper we want to complete and extend the theoretical results in If22l Section 6]. In addition, 
the RKBS denoted in Definition al l is different from IfTTl [T8 23 1. The RKBS redefines in weak sense or 
strong sense and its reproducing kernel K can be set up on nonsymmetric domains, i.e., K : Q2 x£2i — > R 
where O2 an d ^1 can be various subsets of R d (see Definition [47TJ. The RKBS is the extension of the 
RKHS. The RKBS defined in 07] HS1 [23]l can be seen as a special case of the RKBS defined in this 
paper. Even if RKBS is defined in weak sense, we can still obtain the optimal recovery in the weak- 
sense RKBS using the techniques similar as in |f23l (see Lemma |4~Tb . Next we are able to apply this 
optimal recovery to solve support vector machines in RKBSs same as the classical problems in RKHSs, 
i.e., the classical support vector machine in a RKHS H for the training data {(jTj,y ; )} is 

min V L(Xj,yj,f(Xj)) + E(||/|| w ) , 

where the loss function L and S are given. Suppose that & is a weak-sense RKBS with a reproducing 
kernel K defined on £^2 x c R £/ x R d . Under some sufficient conditions of the loss function L and E, 
Theorem 14.21 provide s that the support vector machine in & for the training data {(*/,y/)J ,_j c x C 

N 

min J] L(Xj, y Jt /(*;)) + 2 (||/|| s ) , 
7=1 

has a unique optimal solution Sd,l,i and its dual element is a linear combination of its reproducing kernel 
centered at the given data points X = [x\, . . . ,Xn) £ £2i, i.e., 

N 

k=\ 

It is obvious that the support vector machine in RKBSs is the generalization form of the classical case 
in RKHSs. 

In Section [4!2l we shows how to use a positive definite function <J> to set up different strong-sense 
RKBSs S^(R d ) and with p > 1 whose reproducing kernel is given by K(x,y) = <t>(* — y) (see 

Theorem 14.31 and |4.7| ). It is obvious that S'^(E. d ) is a kind of generalized native spaces. Moreover, 
S^(R flf ) and S'^(H) coincide with the definition of RKBSs given in ll23l . The empirical support vector 
solution in S'^(R. d ) can be represented by the positive definite function O (see Corollary 14.61 ). Theo- 
rem !4. 71 gives an example of the nonsymmetric reproducing kernel K defined on R d x £1 Corollary 14.51 
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and l4.8l provide that these RKBSs can be further imbedded into some Sobolev spaces for some special 
reproducing kernels, e.g., Sobolev spline kernels. 

The Matern functions represent a recent and fast growing research area which has frequent appli- 
cations in approximation theory and statistical learning (see (7][T4)). The papers |9][22) show that the 
Matern functions are positive definite functions and (full-space) Green functions. In Section [5] we use 
the Matern functions to solve the support vector machines in their RKBSs. Let Gg,„ be the Matern 
function with parameter 9 > and degree n > 3d/2. According to our theoretical results, Sq (R. d ) is 
a RKHS and (M, d ) is a just RKBS. Their reproducing kernels are given in the same Sobolev spline 
kernel Kg^„(x,y) := Gg t „(x-y). It is well known that the support vector solution in S^ (R d ) = t Hc Sj ,(W 1 ) 
has the explicit expression 

N 

sd,lz(x) := ^ K 9t „(x, x k ), x e R d , 

k=\ 

(see Theorem l3.U . Here we discover a new fact that the support vector solution in & G (B, d ) has the 
explicit form 

N.N.N 

SD,LX( X ) = ^ c jCkClKgj,, (x + X k ,Xj + X,) , X € R d , 

and many other fresh explicit support vector solutions in the RKBS (R d ) (see Section |5). This 
discovery could produce a new numerical tool for the support vector machines in our future research 
works. 

Remark 1.1. In this paper, the author Ye hope to make corrections of his mistake of the optimal recovery 
of RKBS %(R d ) mentioned in J22 Section 6.2]. Corollary l4~6lis the correction of El Theorem 6.5], 
which is misconception on the dual-element map to be linear. But the main ideas and techniques are the 
same as in his original l22l . The new version of [22] has been updated in Ye's webpage. 

2 Banach Spaces 

In this paper, we review some classical theoretical results of Banach space in IfTTl [T3l [151 [TBI . We 
denote the dual space (the collection of all bounded linear functionals) of a Banach space S to be S' 
and its dual bilinear product as (■, -)®, i.e., 

(/, T) s := T(j), for all T e &' and all / e S. 

ifTBI Theorem 1.10.7] that T>' is also a Banach space. 

If the Banach spaces T>\ and 2?2 are isometrically isomorphic (equivalent), i.e., Si = £2, then we 
say that the both spaces are identical in the sense of their norms in order that their elements can be seen 
to be the same in the both spaces (see IfTBI Definition 1.4.13]). We say that S\ is imbedded into So if 
there exists a positive constant C such that < C H/Hg, for all / 6 Si cg 2 (see Q, Section 1.25]). 

If the Banach space S is reflexive (see IfTBI Definition 1.11.6]), then we have S" = S and (/, g)@ = 
(g, f)<gi for all / e S and all g e S'. For example, the function space L p (Q.; ju) defined on the positive 
measure space (Q, £$n,fi) is a reflexive Banach space and its dual space is isometrically equivalent to 
L 9 (£2;yu) where p,q > 1 and p~ l + q~ l = 1 (see IfTBI Example 1.10.2 and Theorem 1.11.10]). For the 
complex situation, the isometrical isomorphism from L p (Q; /S)' onto L f/ (Q; p) is antilinear. 
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It is well known that we can discuss the orthogonality in Banach spaces with a more general axiom 
system than that in Hilbert spaces. The papers IfTTl [131 IT31 show that every Banach space can be 
represented as an semi-inner-product space in order that the theories of Banach space can be penetrated 
by Hilbert space type arguments. A semi-inner product [•, -]g : £ x S — > C defined on a Banach space 
& is given by 

(0 [f + g,hh=[f,h] s + [g,hh, 

(if) W, gh = *[f, gh, [f, Ag]s = 1[f, g]s, 

(iii) [/,/]* = ll/ll|> 

m u,gh\<[fjMg,gh, 

for all /, g,h e & and all A e C. However, the Heremitian symmetric of the semi-inner product may be 
not true, i.e., [/, g]& + [g,f]<s- This indicates that the generality of the semi-inner product of Banach 
space is a serious limitation on any extensive development parallel to the inner product of Hilbert space. 
We call that / is orthogonal to g in a Banach space S if 

ll/ + ^lla>ll/ll S > for all X eC, 

(see definitions in IfTTl [131 ). Suppose that the Banach space S is smooth (see lfT6l Definition 5.4.1]). 
Using flTl Theorem 2], we can determine that / is orthogonal to g if and only if / is normal to g, i.e., 

[g,f] s = o. 

According to IfTTl Theorem 3] and fl6l Corollary 5.4.18], the following three statements are equivalent: 
(i) S is smooth, (ii) S is Gateaux differentiable (see lfT6l Definition 5.4.15]), (iii) the semi-inner product 
of & is continuous (see IfTTl ). 

We can also obtain the representation theorem in the Banach space by an adaptation of the repre- 
sentation theorem in the Hilbert space. If the Banach space S is smooth and uniformly convex (see lfT6l 
Definition 5.2.1]), then, for every bounded linear functional Tf e there exists a unique / e £ such 
that 

T f (g) = [g,f]B, for allies, 

and ||7/|| s , = \\f\\s- We denote T is the dual element of / and rewrite it as /* := Tf. The dual-element 
map is a one-to-one and norm-preserving mapping from S' onto S. But this map is perhaps nonlinear. 
IfTTl Theorem 7] provides that the semi-inner product of &' has the form [f*,g*]&> = [g,f]& for all 
f*,g* e S'. Let N be a subset of S. We can check that / is orthogonal to N if and only if its dual 
element/* e N ± , i.e., 

[h, fh = (h, f) s = 0, for all h e N. 
For example, the semi-inner product of L^Q; fi) with 1 < p < oa is given by 

\g,fkjptf = ^2— f g(x)f(x)\f(x)r 2 dn(x), for all /,g £1^(0; /4 

WW Jn 

(see examples in IfTTl fT3l ). 
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3 Reproducing Kernels and Reproducing Kernel Hilbert Spaces 



Most of the material presented in this section can be found in the excellent monograph 171 [191 |2TI . 
For the reader's convenience we repeat there what is essential to our discussion later on. 

Definition 3.1 ([21 Definition 10. 1]). Let Q c R d and <H be a Hilbert space consisting of functions / : 
O — > C. *H is called a reproducing kernel Hilbert space (RKHS) and a kernel function K : Q. x £1 — > R 
is called a reproducing kernel for H if 

(0 K(-,y) e <H and (ii) f(y) = (/, K(-,y)y H , for all / e W and all y e Q, 

where (•, •)« is use d to denote the inner product of H. 

Remark 3.1. For simplifying the discussion and proof, we let all the kernel function be real-valued 
and all the function spaces compose of complex-valued functions in this paper. According to lfl6l 
Proposition 1.9.3], it is not difficult for us to extend the theoretical results into complex-valued kernel 
functions or restrict the function spaces into real. 

3.1 Optimal Recovery in Reproducing Kernel Hilbert Spaces 

Theorem 3.1 (representer theorem Ifl9l Theorem 5.5]). Let "H be a reproducing kernel Hilbert space 
with a reproducing kernel K defined on Q. C R'' and £ : [0, oo) — > [0, oo) be convex and increasing. We 
choose the loss function L : M. d X C X C — > [0, oo) such that L(x, y, •) is a convex map for any fixed x G M. d 
and any fixed y € C. Given the pairwise distinct data points X = {xi, . . . ,Xn} £ £2 and the associated 
data values Y = {y\, . . . ,y^} C C, the optimal solution (support vector solution) sd,l£ of 

N 

mmT LiXj,yj,f{Xj)) + Z(\\f\\H), 
feH L r-i 
;=i 

has the empirical representation 

N 

sd,l,z(x) := ^ c k K(x, x k ), x e CI. 

k=\ 

HereD := {{x\,yi) , . . . , (x N ,y N )}. 

3.2 Constructing Reproducing Kernel Hilbert Spaces by Positive Definite Func- 
tions 

Definition 3.2 ( f2~Tl Definition 6.1]). A continuous even function <$> : R d — > R is called positive definite 
if, for all JVeN, all sets of pairwise distinct centers X — {x\ , . . . , x^} c W 1 , the quadratic form 

N N 

^] ^] CjcT®(xj - x k ) = c'A^xC > 0, for all c e C N \{0), 

j=\ k=\ 

/ \ N,N j 

where the interpolation matrix Ao,z := yb(Xj ~ *k)) . ' k _ l 6 C NxN and c* - c . 
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It shows that <t> is positive definite if and only if Ao,z is positive definite for any pairwise distinct 
finite data points X in R d . The application and history of positive definite functions can be seen in the 
review paper JS) . 

Theorem 3.2 (EU Theorem 6.11]). Suppose that O e C(R d ) n Li(R d ). Then O is positive definite 
if and only if O is bounded and its Fourier transform <£> is nonnegative and nonvanishing ( nonzero 
everywhere). 

Remark 3.2. In this paper, the Fourier transform of / e Li(R rf ) is defined by 

f(x) := (2nT d l 2 f me-^dy, 

JR<> 

where i is the imaginary unit, i.e., i 2 = -1. 

II2TI Section 10.2] shows how to use the positive definite functions to construct the RKHSs. 

Theorem 3.3 (|21, Theorem 10.12]). Suppose that O e C(R d ) n Li(R d ) is a positive definite function. 
Then the space 

?4(R d ) := {/ e L 2 (R d ) n C(R d ) : //<l 1/2 e L 2 (R d )} , 
equipped with the norm form 

,2 \l/2 



*(*) 

is a reproducing kernel Hilbert space (native space) with a reproducing kernel given by 

K(x,y):=Q>(x-y), x,y e R d , 
where <£> and f are the Fourier transforms of <t> and f. Its inner product has the form 

(f,g)„ = (2n)- d/2 f^^-dx, f,g€<H*(R d ). 

Using the Fourier transform technique similar as in Theorem 13.31 we can set up RKBSs by the 
positive definite functions (see Section l4!2l ). 



4 Reproducing Kernels and Reproducing Kernel Banach Spaces 

Now we give the definition of RKBSs which is a natural generality of RKHSs by viewing the inner 
product as the dual bilinear product. 

Definition 4.1. Let Oi and Q2 be subsets of R d and S be a Banach space composed of functions 
/ : £l\ — > C, whose dual space &' is isometrically equivalent to a function space f with g : Q.2 — > C. If 
there is a kernel function K : Q2 x Qi — > R such that 

(0 K(-, y) eTsS' and (zz) f(y) = (f, K(-,y)) s , for all/ e S and all je£l h 

then we call S a reproducing kernel Banach space (RKBS) and K its reproducing kernel in weak sense. 
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If the weak-sense RKBS S is further reflexive and has the other-side reproduction, i.e., 



(Hi) K(x, ■)€& and (fv) (K(x, ■), g) s = g(x), for all g e T = £' and all left, 

then it is obvious that T is also a weak-sense RKBS with the RK K(x,y) := K(y, x) defined on Qi x 
^2- With these additional conditions, we say that S is a reproducing kernel Banach space with the 
reproducing kernel K in strong sense. 

Remark 4.1. We know that the Riesz representer map on complex Hilbert space "K is antilinear, i.e., 
Tx g {f) = </, Aghi = W, 8>H = Mf, g)<H = XTtf), for all f,ge<H and all A e C. 

Here we even let the isometrical isomorphism from the dual space &' onto the related function space be 
antilinear. Thus, the format of strong-sense RKBS coincides with the complex RKHS, i.e., 

</, K{-,y)y H = <tf(y, ■),/>„ = (/, K(-,y)), H = f(y), for all / € <H and all yen, 

which indicates that the RKHS is a special case of strong-sense RKBS. 

Why we set up the definition of RKBS different from ||23l Definition 1]? The reason is that we 
can show the optimal recovery in RKBS even if it is defined in weak sense. Moreover, since the dual 
space of Hilbert space is isometrically equivalent to itself, we can choose the equivalent functions space 
T = <H such that the domains of the reproducing kernel K are symmetric, i.e., Q2 = Sl\. Actually the 
Banach space S is usually not equal to any equivalent function space T of its dual S' even though we 
do not require their norms to be equivalent. We naturally do not need the symmetric conditions in the 
Banach space. So the nonsymmetric domains are used to define the RKBS & and the reproducing kernel 
K, i.e., Q,2 ^ Q.i . In this paper, we only consider the nonsymmetric domains chosen in R d . The domains 
of K are related to both S and F = S'. If we choose different T to be isometrically equivalent to the 
dual S', then we can obtain different reproducing kernel K of the RKBS S dependent on its equivalent 
dual space T ■ 

The K(-,y) can be seen as a point evaluation function 6 y defined on S. This implies that 6 y is 
a bounded linear functional on S, i.e., 5 y e S'. If the Banach space S is further uniformly convex 
and smooth, then its semi-inner product and its dual-element map are well-defined. Because the inner 
product is also the semi-inner product and the dual-element map is an identical map on a Hilbert space. 
When S is uniformly convex and smooth, then we can even use the semi-inner product and the dual 
element map to set up the equivalent conditions of weak-sense RKBSs, i.e., 

6 y eS'=T which indicates that f(y) = (/, S,) s = [/, <Jj]s, for all / e & and all y e Q. u 

If S is a strong-sense RKBS, then the equivalent dual space T of S is also a strong-sense RKBS. 
All RKBSs and reproducing kernels set up in Section 1431 satisfy the strong-sense definition but their 
domains can be symmetric or nonsymmetric. 

Suppose that a sequence {f n ]™ = i C £ and / € S such that \\f - f„\\ s — > when n — > 00. We fix any 
y e Cli, Then 

1/00 - fn(y)\ = Kf - fn, K(;y)h\ < \\K(;y)\\ s , 11/ - /„|| s ^ 0, 
when n — > 00. This means that convergence in the weak-sense RKBS £ implies pointwise convergence. 
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Let N be a completion (close) of span \K{-,y) : y e Qi} c T = S' with its dual norm. Now we 
show that N = T = H' which means that span {K(-,y) : y e Qi) is dense in T and {K(-,y) : y e £2i) 
is a linear basis of T when S is a reflexive weak-sense RKBS. It is obvious that N QT- Assume that 
N <iT. According to fW\ Corollary 1.9.7] there is an element / E S = S" = f such that ||/|| s = 1 
and f(y) = (f,K(-,y)) s - for ally e Qi. We find the contradiction between = 1 and / = 0. 
Thus the first assumption is not true and then we can conclude that N = T = S'. 

Example 4.1. We give a simple example of the strong-sense RKBS. Let Q.2 = Qi := [!,••• ,n) and 
A e R" x " be a symmetric positive definite matrix. It can be decomposed into A = VDV r , where D is a 
positive diagonal matrix and V is an orthogonal matrix. We choose p, q > 1 such that p + q^ 1 = 1. 
Define !B := {/ : Oi — > C) equipped with the norm 

||/|| s := ||D- 1/? V7|| 9 , where / := (/(l), ■ • ■ ,f{n)f . 

We can check that & is a Banach space and its dual space S' is isometrically equivalent to f \- 
{g : Q2 — » C) equipped with the norm 

llgll* := ||D- 1/p V r g|| p , where g := (g(l), ■ ■ ■ ,g(n)) T . 

Moreover, its dual bilinear form is given by 

(/. g)s = g'A-'f, for all / 6 S and all ge&. 

If the kernel function is defined by 

K{j,k) := A Jk , jeCh,keni, 
then the reproduction can be easily verified, i.e., 

(f,K(-,k)) s = f(k), keCki, and (K(j, ■), g) B = g(j), j e Q 2 . 
Therefore & is indeed a RKBS in strong sense. 

4.1 Optimal Recovery in Reproducing Kernel Banach Spaces 

It is well-known that the Hilbert space is uniformly convex and smooth. It is natural for us to 
assume the RKBS is further uniformly convex and smooth to introduce its optimal recovery in weak- 
sense RKBS. 

Given the pairwise distinct data points X = {x\, . . . , Xn) £ £2i and the associated data values 
Y = {yu . . . ,y N } c C, we define the subset in the RKBS £ by 

M S (X, Y):={feS: f(xj) = yj, for all j = I, . . . , n) . 

If Ns{X, Y) is a null set, then there is no meaning for the support vector machines. So we need to assume 
that Ns{X, Y) is always non-null for the given data sites. Actually we can show that Ng(X, Y) is non-null 
for any data values Y if and only if 6 Xl , . . . , 5 XN are linearly independent on S because Yit=i c kS Xk — 
if and only if Y!k=\ c kf(Xk) = for all / e & and c = (ci, • • ■ , cn) t = if and only if c orthonormal to 
C N . In this section, we assume that 6 Xl , . . . , S Xfl are always linearly independent on S for the given data 
points X, which is equivalent to the fact that K{-,X\), . . ., K(-,Xn) are linearly independent. 
We use the techniques of [23 Theorem 19] to verify the following lemma. 



8 



Lemma 4.1. Let H be a weak-sense reproducing kernel Banach space with a reproducing kernel K 
defined on Q2 X Qi C M. d X M. d . Suppose that £ is uniformly convex and smooth. Given the pairwise 
distinct data points X — {xi, . . . ,Xm] £ and the associated data values Y — {y\, . . . ,y^} c C, the 
dual element s" D of the unique optimal solution s D of 

min||/|| s s.t. f( Xj )=yjforallj= l,...,N, (4.1) 
is the linear combination of K(-, X[ ), . . . , K(-,x^), i.e., 

N 

s* D (x) = ^ c k K(x, x k ), x e Q. 2 . 

k=l 

HereD := {{x u y\) , . . . ,(x N ,y N )}. 

Proof. We firstly prove the uniqueness of the optimal solution of the minimum problem (14. U . Let us 
assume that the minimum problem ( 14. It has two optimal solutions s\, S2 e & with s\ + s^. Since & is 
uniformly convex, lfl6l Corollary] provides that ||^ (s\ + S2)\\ s < \ Ikills + \ H^lla- Ikills - Italia then 
shows for S3 := | (si + S2) that |ta|| s < Italic and 53 E Ns{X, Y), i.e., s\ is not a optimal solution of the 
minimum problem ( 14. It - The assumption that there are two minimizers is false. 

Because N S (X, Y) + N S (X, {0}) = N S {X, Y) and N S (X, {0}) is a closed subspace of S. We can 
determine that the optimal solution so is orthogonal to Ns(X, {0}), i.e., \\sq + h\\< £ > \\sd\\s f° r a U 
h € Ns(X, {0}). Since S is uniformly convex and smooth, the dual element s* D of so is well-defined and 

[h, s D ] s = (h, s* D ) s = 0, for all h e N S (X, {0}), 

which implies that 

s* D e M S (X, {0}) ± = {g e T = & : (h,g) 3 = 0, for all h e N S (X, {0})} . 
It is obvious that 

N S (X, {0}) = {/£S: f(xj) = (/, K{-, X j))a = 0, for aU j = 1 N} = >an {*(., . 

According to lfl6l Proposition 1.10.15], we have 

s* D e (^span \K(; X k )$ =l ) X = span {K(-, xi), . . . , AT(-, x N )) . 

□ 

If S is uniformly convex and 2 : [0, 00) — > [0, 00) is convex and increasing, then S (||-|ls) is strictly 
convex on S. Now we verify the representer theorem of the support vector machine in the weak-sense 
RKBS. 

Theorem 4.2. Let £ be a weak-sense reproducing kernel Banach space with a reproducing kernel K 
defined on Q 2 x Qi c R d x R d and Z : [0, 00) -> [0, 00) be convex and increasing. Suppose that S is 
uniformly convex and smooth. We choose the loss function L : R X C X C — > [0, 00) such that L(x,y, •) 
is a convex map for any fixed x € R d and any fixed y 6 C. Given the pairwise distinct data points 
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X — {xi, . . . ,Xn} £ £2i and the associated data values Y — {y\, . . . ,y^} c C, the dual element of the 
unique optimal solution ( support vector solution) Sd,lx of 

N 

min J] L( Xj , y h f( Xj )) + S (||/|| s ) , (4.2) 

has the empirical representation 

N 

s *d,lx^ : = X c k K ( x > x k\ x e £l 2 . 

k=\ 

HereD := {{xi,y{) , . . . , (x N ,y N )}. 
Proof. Let 

N 

T D ,Lx(f) ■= Yj L(Xj,yjJ(Xj)) + X(||/U , feS. 

;=i 

Since the convergence in S implies the point wise convergence, we obtain the strict convexity and con- 
tinuity of Tq.lx by the strictly convexity of E(||-|| s ) and the convexity of L(xj,yj, •) for all j = l,...,N. 
The uniformly convex norm indicates its refiexivity by Milman-Pettis Theorem Ifl6l Theorem 5.2.15]. 
According to the existence of minimizers theorem |fl9l Theorem A.6.9], Td,lx has a global minimum 
because S is a reflexive Banach space. 

We fix any / e S. According to Lemma |4~T1 there exists an element s/ t x whose dual element 
s*j: X e span{K(-,Xk)} k= i such that Sfjc interpolates the data values {f(xi), . . . ,f(xN)} at the centers 
points X = {xi, . . .,Xn} and ||*/,x|| s ^ ll/lls- This implies that 

T D ,L£(Sfjd ^ T D,L,?.(f)- 

Therefore the dual element s* D L z of the optimal solution sqxx of the minimum problem ( 14.2b belongs 

to Spw{K(;X k )}* =1 . 

□ 

Remark 4.2. Since K(-,xj) can be seen as a point evaluation function 6 X defined on S, it indicates 
that the dual element of sd,lx can be also written as a linear combination of 6 Xl , . . . , 6 Xn , i.e., s* D L „ = 
Ef=i CjS Xj . 

The uniform convexity and smoothness of & indicates the uniform convexity and smoothness of its 
dual S' = T . If S is a strong-sense RKBS, then we can further obtain the optimal recovery in f in the 
same way, i.e., the dual element of the optimal solution (support vector solution) of 

N 

SFL Yj L ( x j>yj>8(Xj)) + 2(llgll s , 
ger='B' *—f 

is the linear combination of K{x\, ■),..., K(xn, •), where the data points X = {x\, . . . ,x^} Q Q 2 . 

Moreover, since the dual-element map is an identical map on the Hilbert space and the reproducing 
kernel of RKHS is symmetric, the optimal recovery in RKBS as in Theorem 14.21 can be seen as the 
generalization form in RKHS as in Theorem l3.ll 
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4.2 Constructing Reproducing Kernel Banach Spaces by Positive Definite Func- 
tions 



Now we construct the RKBS by the positive definite function in a similar way of RKHS as The- 
orem[33] Let 1 < p,q < oo and p~ l + q' 1 = 1. Suppose that $ e C(R d ) n Li(R d ) is a positive 
definite function. According to Theorem 13.21 we know that <I> € L 1 (R flf ) n C(R d ) is nonnegative and 
nonvanishing. We define 

#£(R d ) := {/ e C(R'') n SI : the distributional Fourier transform / of/ 

is a measurable function defined on R d such that //4> 1/,? E L 9 (R £/ )} , 

equipped with the norm 

(2ny d/2 I 'V '' dx 



J <£(*) 



where SI is the collection of all slowly increasing functions (see ||2"T1 Definition 5.19]). We define 
&^(R d ) in an analogous way as above. 

Remark 4.3. Following the theoretical results of lfl2l Section 7.1] we can define the distributional 
Fourier transform T e 5?" of the tempered distribution T e S^" by 

(y.t)s' :=<?,ZV, for all ye^, 

where is the Schwartz space (see ETl Definition 5.17]) and 5?' is its dual space with the dual bilinear 
form <■, We can also verify that C(R fl ') n SI c L' 1 oe (R rf ) n SI is embedded into S" . 

Suppose that q < p. Since 4> E Li(R d ) n C(R d ) and p/q > 1, 4W« E Li(R d ) which will be used 
in the proof of the following theorem. One additional symmetric condition of <b q ' F E L^R' 5 ') is also 
needed in the proof. 

The positive measure p on R d is well-defined by 

r dx 

p(A) := (2ny d/2 , for any open set A of R d . 

Ja <*>(*) 

So the space Lg(R d ;p) is well-defined on the positive measure space {R d , ^^i,p), i.e., 
L 9 (R rf ; p) := If : R d -» C : /is measurable and [ |/(x)| 9 d//(jc) < oo 
equipped with the norm 

\\f\\L^; M) -=U\f(x)\ q dp(x)\ . 

L p (R d ; p) is defined in an analogous way. It is well-known that L ? (R d ; p) is a Banach space and its dual 
space ~L q {R d ; p)' is isometrically equivalent to L p (R d ; p). In the same way of representation theorem on 
Hilbert space, the bounded linear functional T g e h q (R d \ p)' associated with g e L p (R d ; p) is defined by 



-j 

J J: 



T S (J):= I f{x)g{x)dp(x), for all /eL^;//) 
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Here, this isometrical isomorphism from L 9 (R fl '; p)' onto L p (R d ; p) is antilinear same as the dual of 
complex Hilbert spaces, i.e., 

7W)= f f(x)I£x)d(x(x) = AT g (f), forall/eL ? (R d ;j u)andalUeC. 

If we can show that S^(R fl ') and L 9 (R rf ; p) are isometrically isomorphic, then !B P tl iR d ) is a reflexive 
Banach space and its dual space S'^(R d Y is isometrically equivalent to L p (R d ; p). It is analogous to 
&l(R d ) = L p (R d ; p). If we can further check the both sides reproduction of S^(R d ), then & p (S> (R d ) is a 
strong-sense RKBS. 

Theorem 4.3. Let I < q < 2 < p < oo and p~ x + q~ l = 1. Suppose that <5 E L 1 (R ') n C(R d ) is a 
positive definite function on R d and that O q ' p 6 Li(R c '). Then S^(R d ) is a strong-sense reproducing 
kernel Banach space with a reproducing kernel 

K(x,y) := <D(jc - y), x,y e R d . 

Its dual space S'^{R d )' andS q ^(R d ) are isometrically isomorphic. Moreover, S^{R d ) is uniformly convex 
and smooth. In particular, when p — 2 then Sj ) (R ') = ( Ha>(JsL d ) is a reproducing kernel Hilbert space 
same as in Theorem\3.3\ 



Proof. We firstly prove that S'^(R d ) and L f/ (R flf ; p) are isometrically isomorphic. The Fourier transform 
map can be seen as a one-to-one map from S'^(R d ) into L q (R d ; p). We can check the identity of their 
norm 



So the Fourier transform map is an isometric isomorphism. Now we prove that the Fourier transform 
map is surjective. Fix any /; e L q (R d ; p). We want to find an element in S'^(R d ) whose Fourier transform 
is equal to h. We conclude that h e L{(R d ) because 

\h(x)\dx < I f ^^-dx) 7 f ®(x) p/ci dx\ ' < oo. 

\JR>> ®(x) j \J9J I 

Thus, the inverse Fourier transform of h as h(x) = (2ny d ^ 2 J Rd h(y)e' xTy dy is well-defined and an element 
of C(R d ) n SI. This shows that h e %(R d ) and h = h because (h,j)y = {h,y).y = {h,y).y for all 
y e 5? . Therefore S'^(R d ) is isometrically equivalent to L ? (R rf ; p). 

Using e L^R'') we can also prove that (R d ) = L p (R d ; p) in an analogous way. Therefore 
£^(R d ) is isometrically equivalent to the dual space of S^iR 4 ). 

We fix any y e R"'. The Fourier transform of K(-,y) is equal to k y (x) :- <h{x)e~ lx> y . Since <b p/q e 
U(R d ) we have k y e L p (R d \p). Thus K(-,y) can be seen as an element of &l(R d ) = S^(R d )'. In 
addition, we also have K(x, •) e S'^(R d ) because k x e L q (R d ; p) for any x e R d . 

Finally we verify the reproduction. Fix any / e S^iR 1 ) and j> e R d . We can verify that / e Li(R rf ) 
as in the above proof. Moreover, the continuity of / and / allows us to recover / pointwise from its 
Fourier transform via 

f(x) = f(x) := i2nY dil f fiyy^dy. 
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Thus, we have 



= {2nT dl2 f f{x)e ixT y&x = fiy). 
In the same way, we can also verify that &^(R d ) has the other-side reproduction, i.e., 



(K(x, •), g) Km = (k x , g\ {W , v) = g{x), for all g e <B% (R d ) = %(R d )' and all x e R d . 

Therefore %(R d ) is a strong-sense RKBS. 

Since L q (R d \p) is uniformly convex and smooth by lfl6l Theorem 5.2.11 and Example 5.4.8], 
S'^{R d ) = L q (R d ;p) is also uniformly convex and smooth. □ 

Remark 4.4. It is obvious that S^{R d ) is a also a RKBS with the reproducing kernel K. Combining with 
lfl6l Proposition 1.9.3], the restriction of S'^(R d ) in real is also a RKBS with the reproducing kernel K 
and its dual is isometrically equivalent to the restriction of S^(R d ) in real. It is well-known that the 
RKHS of the given reproducing kernel is unique. But Theorem l4. 31 shows that the reproducing kernels 
of different RKBSs can be identical. 

Corollary 4.4. Let &l(R d ) with p>2be defined in Theorem\4J\ Then %(R d ) Q L p (R d ). 

Proof. We fix any / e S^(R d ). According to the proof of Theorem l4.3l we have / e L 9 (R rf ) because 

f \f(x)\ q dx < (2n) qd/2 [ f !£^-djc) (sup €>(*)) < oo. 
Jr-' [J:- <\Hx) jW / 

The Hausdorff- Young inequality lfl2l Theorem 7.1.13] provides that f - f e L p (R d ) because 1 < q < 
2. a 

Remark 4.5. The reproducing kernel Banach space S'^(R d ) can be precisely written as 

&l(R d ) := {/ € L p (R d ) n C(R d ) : the distributional Fourier transform / off 

is a measurable function defined on R d such that //€> I/9 e L 9 (R d )} . 

However, (R"') <£ L 9 (R d ) because the Hausdorff- Young inequality does not work for p > 2. 

We fix any positive number m > d/2. According to fl2Tl Corollary 10.13], if there are two positive 
constants C\, Ci such that 

Ci (l + \\x\$f m ' £ < <i>(*) 1/2 < C 2 (l + ||jc|i) 



'" /2 ^-' nl \ x€R d 



then the RKHS £ 2 (R d ) = 74^) and the classical L 2 -based Sobolev space W"(R d ) = f H"'(R d ) of 
order m are isomorphic, i.e., < H<s,(R d ) = < H m (R d ). 

Following the ideas of RKHS, we can also find the relationship between the RKBS and the Sobolev 
1 + IMI2) /W w ith P - 2- The theory of singular integrals then shows that / 
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belongs to the classical L^-based Sobolev space W'p(R d ) of order m if any only if the function f m is the 
Fourier transform of some function in h p (R d ), and the L p -norm of the inverse Fourier transform f m is 
equivalent to the W™-norm of / (much more detail mentioned in JT] Section 7.63] and |[T2l Section 7.9]). 
Using the Hausdorff- Young inequality, we can set out 11/11^^) < C \\fm\\ Lm ^ C ||/ m || L(;(R ^ for some 
positive constant C independent on /. Following these statements, we can introduce the following 
corollary. 

Corollary 4.5. Let %(R d ) with p > 2 be defined as in Theorem \4.3\ and W'p(R d ) be the classical L p - 
based Sobolev space of order m > pd/q — d/q. Here q is the conjugate exponent of p. If there are two 
positive constants C\, C2 such that 

Ci (l + IWIi)"" /2 < 0(*) 1/? < C 2 (l + IWIi)"" /2 , xeR d , 

then %(R d ) is imbedded into W' p n (R d ), i.e., 

H/llw^) < C \\f\\ Km , / e ££(R d ) c Wp"(R d ), 

for some positive constant C independent on f. 

Remark 4.6. Here the lower bound of m is induced by the condition of <P* ,p e Li(R f/ ). According to 
Corollary 14.51 the dual space W q "'{R d ) of the Sobolev space W' p "(E. d ) is imbedded into the dual space 
S^(R d )' of the RKBS S^R" 1 ). It is well-known that the point evaluation function S x belongs to W q "\R d ) 
(see E Section 3.25]) which coincides with 6 X e %(R d )'. 

Since K(-,X\), . . . ,K(-,Xn) are linear independent in S^(R d ) 2 S£(R fl ')' for any pairwise distinct 
data points X = [x\, . . . , Xn] £ R d , 8 Xi , . . . , 5 XN are linearly independent on S'^{R d ). According to 
Theorem |4.21 we can solve the support vector machines in &^(R d ) with p > 1, Since the dual element 
°f s *dli. * s ec l ua l t0 s d,l,z, we can also obtain the empirical support vector solution. 

Corollary 4.6. Let &^(R d ) with p > 1 be defined as in Theorem \4.3\ and 2 : [0, 00) — » [0, 00) be convex 
and increasing. We choose the loss function L : R d X C X C — > [0, 00) such that L(x, y, •) is a convex map 
for any fixed x E R d and any fixed y € C. Given the pairwise distinct data points X — {x\ , . . . , Xn] £ R d 
and the associated data values Y — {yi, . . . ,yn] C C, the unique optimal solution (support vector 
solution) Sd,l,t of 

N 

jseu Tj yj> K*^ + s . 

has the empirical representation 

p-2 

dy, x e R d , 

where q is the conjugate exponent of p and D :— {(xi, y{) , . . . , (xm, yjv)}. 
Proof. We know that the dual element of / e L p (R d ; p) = L q (R d ; p)' is given by 

\\f\f 

UJ u L p (BL d ;p.) 

Using the Fourier transform and the inverse Fourier transform, we can get the empirical form of the 
optimal solution Sb,lx- d 



Sd,l,i.(x) = (2n) 



-d/2 



r N 

&(yy /<l Yc k e 



i(x-x0y 



z 

k=\ 



c k e 



■ix. y 
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Remark 4.7. In particular, if p is an even positive integer, then sd,lx is also a linear combination of 
some kernel function transposed at the data points X. For example, when p — 4, then 

N,N,N N,N,N 

sd,l,h(x) = 2^ c f~kCi$>i (x - Xj + x k - */) = ^ cjcJciK 3 (x + x k ,Xj + */) , x e R d , 

j,k,l=\ j,k,l= 1 

where the kernel function Ky,(x,y) := <&t,(x - y) and (£3 is the inverse Fourier transform of O 3 . Moere- 
over, 

S D,L,z\\ B ^ (Rd) = [«£),L,E, ■SD.L.e]®^^) = <«D,L,E, ■S £ ),L,s)sj0R'') 
Af N,N,N,N 
= ^C~<S DjLjI ,#( s A:y)) S £p l <, ) = ^ C^CiQC„A" 3 (*/ + X k , Xi + JC„) . 
j=\ j,kj,n=\ 

In this section, we can think that the RKBS can be given in the /?-norm, where 1 < p < oa. The 
generalized Sobolev spaces in J6l O can be further set up with the 1-norm. But it is well-known that 
the 1-norm space is non-uniform-convex, non-smooth or even non-reflexive. So we can not define a 1- 
norm RKBS in strong sense which indicates that the 1-norm RKBS does not fit the definition of RKBSs 
given in ||23ll also. Moreover, the optimal recovery of the support vector solutions may be not true for 
the 1-norm cases because there may be no semi-inner products induced in the 1-norm RKBSs. The 
papers IfTTl |T8l show that we can solve the support vector machines in some special 1-norm RKBSs 
with the typical reproducing kernels. We will try to use the idea of the weak-sense RKBSs to construct 
the 1-norm RKBSs and solve the support vector machines for the general 1-norm cases in our future 
research. 

We use the techniques of |3 , Theorem 6] to set up a strong-sense RKBS defined on a subset Q of 

R d . 

Theorem 4.7. Let S^(R. d ) with p > 1 be defined as in Theorem 14.31 and Q c R d . Then the function 
space 

S^(Q) := \li : there exists a function h e f^ 1 (R fl ') such that /|n = /zj , 
equipped with the norm 

\\h\y, a) := min s.t. f\ n = h, 

is a strong-sense reproducing kernel Banach space with the reproducing kernel 

K(x,y) := <b(x-y), x e R d , y e Q, 

where f\a stands for the restriction of f into Q. Its dual space S^(Q)' is isometrically equivalent to a 
closed subspace of23^(R d ) (the annihilator of No in S^(R d )) 

Atf = [g e %(R d ) ee %(R d Y : (/, g) %(m = 0, for allfeNo}, 

where q is the conjugate exponent of p and 

N :={fz%(R d ): /b = o}. 

Moreover, S^(O) is uniformly convex and smooth. 
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Proof. Since the convergence in the RKBS T> p <Sl (R d ) implies that the point wise convergence, we can 
determine that No is a closed subspace of S'^(R d ). According to the construction of S^(Q), S^(Q.) is 
isometrically equivalent to the quotient space S^(M. d )/No (see |fT6l Definition 1.7.1 and 1.7.3]). Thus 
S^(C2) is a reflexive Banach space by Ifl6l Theorem 1.7.9 and Corollary 1.11.19]. 

Next we use the identification of (s^(R d )/Noj = Aft to verify the reproduction (see lfl6l Theo- 
rem 1.10.17]). We fix any_y e Q. Since 

</, K(-,y)) %m = f(y) = 0, for all / € N , 

we have K(-,y) e Aft = (s p ^(R d )/N )' = Combining this with the reproduction of %(R d ), we 

have 

(h,K(;y)) %(n) = (Eh,K(-,y)) %m = (Eh)(y) = h(y), 

for all h e S^(£2) and all y e Q,, where E is the extension operator from into S'^(R d ) such that 

Eh\a = h and ||£&|ls£(R<i) = PHs?(q). Since K(x, -)ln e for all x € R d , we can also obtain the 

other-side reproduction of i.e., 

{K(x, -)ln, g>£j(£i) = {K(x, -),g) s ^ m = g(x), 

for all g e Aft = S£(f2)'. Therefore S'^(Q.) is a strong-sense RKBS with the reproducing kernel A"|r</ X £2- 
Since S p JR d ) is uniformly convex, OH Theorem 5.2.24] provides that = S^(R d )/No is 

uniformly convex. We also know that S^iW 1 )' = L q (R d ; fi) is uniformly convex and Aft is a closed 
subspace of £|(R d ) = %(R d Y by HU Proposition 1.10.15]. Combining with 02] Proposition 5.1.20 
and 5.4.5], we can also check that S^(f2) is smooth. □ 

Remark 4.8. When p = 2, then we know that S% (Q) is a Hilbert space by Theorem l4.3l Thus its dual 
space and itself are isometrically isomorphic such that its reproducing kernel becomes K\q x &. Since 
Si (R d ) = No © Aft, we can determine that [g\ n : g e Aft) = S|(£l) and llglla^ = llgbllsj OT for all 
g e Aft which implies that Sj,(Q) = Aft s S^(Q)' and S^(0) has the inner product 

(/zi,/z 2 ) S | (n) = </Ji,ft2>s|(n) = (Eh u Eh 2 ) S 2^ Rd) = (Eh u Eh 2 ) S 2 >m , 

for all /2i,/i 2 e S^(O). Therefore S^(Q) is a RKHS. Moreover, since A"(-,jO e Aft for any y e Q, we 
have E (K(-,y)\o) = K(-,y). This shows that K\q x q is a reproducing kernel of S^iQ). This conclusion 
is the same as in (3J Theorem 6], 

If the RKBS is even a Hilbert space, then we can choose an equivalent function space of its dual 
as itself such that its reproducing kernel has symmetric domains. The difficulty to find an equivalent 
function function space of the dual of RKBS, which is defined on the same domain of the RKBS, causes 
the domains of its reproducing kernel to be nonsymmetric. Theorem l4.3l and l4~7l provide us the examples 
of symmetric and nonsymmetric reproducing kernels of RKBSs, respectively. 

If the subset Q is a regular domain, then the definition of weak derivatives (see Q] Section 1.62]) 
provides that f\ n e W™(Q) and \\f\n\\ w;m) < \\f\\w;m<) for a11 / 6 W™(R d ), where W™(Q) is the L p - 
based Sobolev space of order m. Now we use the imbeddings of 2?£(R rf ) to derive the imbeddings of 
#£(Q). We fix any h e According to CorollarygjS] we have 

IWIw-<H) < \\Eh\\w;<v>) < C \\Eh\\^ m = C \\h\\ K(n) , h e BftO) c W' p "(D.), 
for some positive constant C independent on h. 
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Corollary 4.8. Let S£(R d ) with p > 2 and m > pd/q — d/q be defined in Corollarv \4.5\ Here q is 
the conjugate exponent of p. Suppose that D. C R rf is regular. Then S^(Q.) defined in Theorem \4.7\ is 
imbedded into the L p -based Sobolev space of order m, i.e., 

PIIw-ct < C \\h\y o(n) , h e ££(Q) c W" p \Q.\ 

for some positive constant C independent on h. 



5 Examples for Matern Functions 

||9] Example 5.7] and Il22l Example 4.4] show that Matern functions (Sobolev splines) with shape 
parameter 8 > and degree n > d/2 

2l-n-d/2 

is a positive definite function on R d , where t i-> AT y (?) is the modified Bessel function of the second 
kind of order v and f i-» F(f) is the Gamma function. Moreover, Ge, n is a full-space Green function of 
differential operator Lg _„ : = (fl 2 / - A) , i.e., Lg^Ggj, = Sq. The Fourier transform of Gg, m has the form 

Gg, n ( X ) = (# 2 + , X € R d . 

Let \ <q<2<p<oo with p _1 + g -1 = 1 such that nq/p > d/2 and m := 2n/q. Since G^ e 
Li(R d ), Theorem 1431 provides that (R rf ) is a RKBS on R rf with a reproducing kernel J^Qcy) = 
Ge, n {x ~ y)- We can also check that there are two positive constants C\ , C2 such that 

Ci (l + \\x\\ 2 ^) ml2 < Gg,Ax) llq < C 2 (l + l|^|||)" m/2 • xeR d , 

According to Corollary 03J and gj?] the RKBS (R d ) is imbedded into W™(R d ) and the RKBS 
(Q.) is imbedded into for any regular domain Q. of R d . 

In particular, we let p := 4. Then G? = G^,,. According to the discussion of Corollary 14.61 and 
Remark l4~7l the optimal solution of the support vector machine 



N 

min YL(x j ,y j ,f(x j )) + I.(\\f\\ si m ), 



has the empirical representation 

N,N,N N,N,N 
SD,LX(X) = c fkClG 6M {x - Xj + X k ~Xi)= ^ CflaKejn (x + X k ,Xj + X,), X 6 R d , 

j,k,l=l j,k,l=\ 

and its coefficients c = (c\ , • • • , c^) T are solved by the following minimal problem 



N 



( N,N,N \ ( N.N, N.N \ l l 2 



lin ^ L x i>yi> 2 CjC k CiK e ,3„(xj + X k ,Xj + X/) +E 2] C,-C_/CiC/^,3„(xi +Xj,X k + */) 



min 

i=l V j,fe,/=l y \i,j,k,l=i 
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where the loss function L and S are defined as in Corollary 14.61 More general, when p — 21 is even, 
then the support vector solution sd,l,e m ^ s a linear combination of the product groups of the 

reproducing kernel bases, i.e., 



sd^(x) = 2^ Y\ ck2 ' i Y\ cki ' Ke - ipi) " 



teS?5, 7=1 



7=1 



7=1 



where k :- (k u --- ,k p ^) T andSfj^ := {it e N'^ 1 : 1 <kj < N for all j = 1, . . . ,p - l}. 

The Matern functions has been applied into the fields of statistical learning (see fBl ). The new 
discover of Matern functions could help the people to create a new numerical tool for support vector 
machines in RKBS. 



6 General Support Vector Machines 

In this paper, we can review the empirical support vector machine ( 14.21 ) by the empirical measure 
'■= lZk=i ^(*a,vi) °f the observations data D :- {(x\,y\) (XN,yN)}, i-e., 



mm 

fe'B 



f L(x,y,f(x))d¥ D (x,y) + Y(\\f\\ s ). 

Jn,xc 



We can generalize the empirical measure to the continuous probability measure Pc defined on Qi x C, 
e.g., the classification probability l?c(x,y) := rj(y\x)p(x), where p is a continuous probability measure 
on Q.\ and rj(-\x) is a classification conditional probability measure on C given by 



7](y\x) := 



k(x), y=l, 
1-k(x), y = -l, 
0, otherwise, 



with k(x) e [0, 1] for all x € Q,\. We can also use this probability measure Pc to set up a general support 
vector machine, i.e., 



mm 

fee 



in f L(x,y,f(x))d¥ c (x,y) + -L(\\f\\ s ). 

' B Jn,xc 



In our future research work, we will try to obtain the representation of general support vector solutions 
similar as in Ifl9l Theorem 5.8]. Its numerical experiments will appear in our later research papers. 
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